Design of the MUC-6 Evaluation
نویسندگان
چکیده
The sixth in a series of "Message Understanding Conferences", which are designed to promote and evaluate research in information extraction, was held last fall. MUC-6 introduced several innovations over prior MUCs, most notably in the range of different tasks for which evaluations were conducted. We describe the development of the "message understanding" task over the course of the prior MUCs, some of the motivations for the new format, and the steps which led up to the formal evaluation. 1
منابع مشابه
Statistical significance of MUC-6 results
The results of the MUC-6 evaluation must be analyzed to determine whether close scores significantl y distinguish systems or whether the differences in those scores are a matter of chance. In order to do such an analysis , a method of computer intensive hypothesis testing was developed by SAIC for the MUC-3 results and has been use d for distinguishing MUC scores since that time . The implement...
متن کاملEvaluation of Coreferences and Coreference Resolution Systems
Reference Resolution (or coreference) is an important tool for information extraction systems (systems which extract facts regarding \who did what to whom, when, and where"). Because of its importance to information extraction, coreference was added as a formal task for the Sixth Message Understanding Conference (MUC-6) MUC-6] held in 1995. MUC-6 was successful in directing the attention of nat...
متن کاملGE adjunct test report: object-oriented design and scoring for MUC-4
This paper reports on the results of the adjunct test performed by GE for the MUC-4 evaluation of text processing systems . In this test, we evaluated the effect of an object-oriented templat e design and associated matching conditions on the scores . The results indicate that the curren t MUC.{ "flat" templade design with cross-references closely approximates a true object-oriente d design . H...
متن کاملOverview of Results of the MUC-6 Evaluation
The latest in a series of natural language processing system evaluations was concluded in October 1995 and was the topic of the Sixth Message Understanding Conference (MUC-6) in November . Participants were invited to enter their systems in as many as four different task-oriented evaluations . The Named Entity and Coreference tasks entailed Standard Generalized Markup Language (SGML) annotation...
متن کاملText filtering in MUC-3 and MUC-4
One of the changes from the Third (MUC-3) to the Fourth (MUC-4) Message Understanding Conference was the emergence of text filtering as an explicit topic of discussion. In this paper we examine text filtering in MUC systems with three goals in mind. First, we clarify the difference between two uses of the term ~text filtering ~ in the context of data extraction systems, and put these phenomena ...
متن کامل